Method and system for fast access to advanced visualization of medical scans using a dedicated web portal

ABSTRACT

A system for viewing at a client device a series of three-dimensional virtual views over the Internet of a volume visualization dataset contained on centralized databases employs a transmitter for securely sending volume visualization dataset from a remote location to the centralized database, more than one central data storage medium containing the volume visualization dataset, and a plurality of servers in communication with the centralized databases to create virtual views based on client requests. A resource manager load balances the servers, a security device controls communications between the client device and server and the resource manager and central storage medium. Physically secured sites house the components. A web application accepts at the remote location user requests for a virtual view of the volume visualization dataset, transmits the request to the servers, receives the resulting virtual view from the servers, and displays the resulting virtual view to the remote user.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/116,524 filed Aug. 29, 2018, which is a continuation of U.S. patentapplication Ser. No. 15/681,785 filed Aug. 21, 2017, now U.S. Pat. No.10,084,846, which is a continuation of U.S. patent application Ser. No.15/255,775 filed Sep. 2, 2016, now U.S. Pat. No. 9,749,389, which is acontinuation of U.S. patent application Ser. No. 14/754,953 filed Jun.30, 2015, now U.S. Pat. No. 9,438,667, which is a continuation of U.S.patent application Ser. No. 14/187,996 filed Feb. 24, 2014, now U.S.Pat. No. 9,106,609, which claims priority from U.S. patent applicationSer. No. 14/322,359 filed Nov. 23, 2011, now U.S. Pat. No. 8,701,167,which is a National Stage Application of PCT/US2010/036355 filed May 27,2010, which claims priority from U.S. provisional application No.61/181,695, filed on May 28, 2009.

BACKGROUND OF THE INVENTION 1. Field of the Invention

This invention relates to a method and system for viewing at a clientcomputer a series of three-dimensional virtual views, transmitted overthe Internet, of a volume visualization dataset contained on one or morecentralized databases.

2. Description of Related Art

The demand for easy access to medical scans is increasing and theexpectation for a simpler and faster way to interpret these large scansis growing. Technology exists to present richer three-dimensional (3D)views from existing two-dimensional (2D) scans that may lead to betterdiagnosis and prognosis as well as improved patient care. However, thecurrent solutions are impractical for 3D to be ubiquitous.

Computed Axial Tomography (CT) scans or magnetic resonance imaging (MRI)scans of a patient's body results in large 3D volume datasets that aretime consuming when transported over the Internet. The scans aretypically spaced two-dimensional planar cross-sections of the patient'sbody or a portion thereof, such as an organ. The scans of these or otherobjects may be stored as a volume visualization dataset in an otherwiseconventional data storage medium accessible by a computer or otherspecialized processor. Assuming the scans are parallel images arrangedin the sequence in which they are found in the scanned object, 3Dvirtual views of the volume visualization dataset are made by selectinga plane to be cut through the volume of the object at a particularlocation and angle. The selected plane may be parallel to the scans, orat any angle to the scans. The location may be anywhere within thescanned volume of the object. The 3D virtual image is a two-dimensionalrepresentation of the 3D object showing the desired perspective of viewand may include images showing depth of or through the object in adirection normal to and behind the selected plane of view.

Generating dynamic 3D views requires processing of raw 2D scans into 3Dand to isolate feature of interest from the raw scan. Therefore, for 3Dto be available, either the user's computer or a dedicated server needsto be powerful enough to support this processing power and the 2D scansneed to be directly available to the user's computer via a high speedcommunication link. Facilities such as Hospitals and Imaging Centersneed to provide the software, hardware, and networking infrastructureand support the IT administration to allow their physicians access to3D. This becomes expensive and adds an administrative burden that only aselect few provide such a “luxurious” functionality to their physicians.The present invention overcomes this limitation by teaching a method andsystem of a common and centralized infrastructure, for receiving,storing, processing and viewing large medical scans via a low-bandwidthweb portal where economics of scale can be applied generously.

Internet access via web portal particularly low bandwidth and highlatency situation of rural area can hinder the user interactivity ofinformation presented in 3D. Latency, i.e., delay in the time it takesfor a packet of data to travel from one designated point to another inresponse to a request, is particularly problematic for users having lowbandwidth Internet communication. Unless otherwise noted, the term “lowbandwidth” refers to communication speed of no more than about 1.5 Mbps,and the term “high latency” refers to response time of more than about125 ms. The user may request arbitrarily and dynamically generatedviews. This requirement normally mandates that raw scan data needs to bepresent on the user's computer for manipulation. For large 3D,four-dimensional (4D), or higher dimensional scans, the retrieving orraw scan for processing is impractical or impossible over low bandwidthnetwork. (Scans in 4D can come in the form of time varying 3D scans, oras a result of combining two datasets, such as PET and CT scannersdataset combined in one. Also, when PET-CT are combined and have a timecomponent, this can be termed a 5D scan, which involves very largedatasets.) The publication by Klaus Engel on Remote 3D VisualizationUsing Image-Streaming Techniques teaches how volume visualization can beobtained via a client and server interaction over the web without havingto retrieve raw scan on to user machines. However, such techniquerequires sufficient bandwidth and low latency network to presentinformation at interactive frame rates. This presents a problem forrural areas to gain access to medical scans in advanced visualizationformat interactively.

Furthermore, medical service providers are required to keep medicalrecords for six years or more and have to do so with utmost security andprivacy. This becomes expensive and adds a management burden tofacilities as there can be IT, physical space and cost administrationlimitations. The end effect is that a facility's operation remains an adhoc process.

SUMMARY OF THE INVENTION

Bearing in mind the problems and deficiencies of the prior art, it istherefore an object of the present invention to provide one or more ofthe following: 1) a method and system that overcomes low bandwidth andhigh latency limitations that are inherent properties of the standardInternet and permits bandwidth usage to be optimized, particularly forinteractive web application for visualizing large medical scans andother volume visualization datasets on any Internet connection; 2) amethod and system that overcomes the problems of maintaining medicalrecords for long periods of time under security and privacy by providinga common and centralized infrastructure for receiving, storing,processing and viewing large medical scans via a web portal whereeconomic of scale can be applied generously.

Still other objects and advantages of the invention will in part beobvious and will in part be apparent from the specification.

One or more of the above and other objects, which will be apparent tothose skilled in the art, are achieved in the present invention which isdirected to a system for viewing at a client device a series ofthree-dimensional virtual views over the Internet of a volumevisualization dataset contained on at least one centralized databasecomprising at least one transmitter for accepting volume visualizationdataset from remote location and transmitting it securely to thecentralized database, at least one central data storage mediumcontaining the volume visualization dataset, and a plurality of serversin communication with the at least one centralized database and capableof processing the volume visualization dataset to create virtual viewsbased on client request. The system also comprises a resource managerdevice for load balancing the plurality of servers, a security devicecontrolling the plurality of communications between a client device, andthe server; and between the resource manager and central storage medium,and at least one physically secured site for housing the centralizeddatabase, plurality of servers, at least a resource manager, and atleast a security device. The system further comprises a web applicationadapted to satisfy a user's request by accepting at a remote location atleast one user request for a virtual view of the volume visualizationdataset, transmitting the request to at least one of the servers,receiving the resulting virtual view from the at least one server, anddisplaying the resulting virtual view to the user at the remotelocation.

The system secured site may be a SAS70 Type II Compliant or equivalentaudited or tested data center. Multiple secured sites located ondisparate physical location are connected via an intranet.

The web application may select a secured site by using a domain nameaddress on a web browser and/or a secured site IP address of a resourcemanager by Domain Name System. The web application may be a medicaldevice for diagnostic imaging viewing purpose. The web application maybe authenticated using an authorized user credentials prior totransmitting the request to at least one of the servers.

The authenticated web application may list the volume visualizationdatasets that are viewable by the authorized user, and may initiate avirtual viewing by selecting a volume visualization dataset. The requestfor a virtual view of the volume visualization dataset may employ an XMLdocument or web server on Hypertext Transfer Protocol Secure (HTTPS).

The web application may communicate with the resource manager device,and may be allocated with at least one server by the resource managerdevice. The web application is adapted to commence interaction with theserver to satisfy user requests, and may be an Active-X component, aJava applet or JavaScript, Flash or Silverlight or other web browserscriptable language.

The system may further include an encrypted communication connectionover which the request and the resulting virtual view are transmitted.The encrypted communication connection may employ SSL/TLS and mayfurther include secure communication ports that communicate only withSSL/TLS turned on.

In another aspect, the present invention is directed to a method forviewing at a client device a series of three-dimensional virtual viewsover Internet of a volume visualization dataset contained on at leastone centralized database. The method comprises providing at least onecentral data storage medium containing the volume visualization dataset,providing at least one server in communication with the at least onecentralized database and capable of processing the volume visualizationdataset to create virtual views based on client request, and providing aclient device linked to the at least one server and central storagemedium over the Internet, the client device having local data storagemedium for storing frames of views of the volume visualization dataset.The method then includes requesting at the client device at least onethree-dimensional virtual view of at least a portion of the volumevisualization dataset, determining if any frame of the requested atleast one view of the volume visualization dataset is stored on thelocal data storage medium, and sending from the client device to theserver a request for any frame of the requested at least one view notstored on the local data storage medium. The method further includes, atthe server, creating the requested frame of the requested at least oneview from the volume visualization dataset in the central storagemedium, transmitting the created frame of the requested at least oneview from the server to the client device, and displaying the requestedat least one three-dimensional virtual view of the volume visualizationdataset at the client device by displaying either the frame transmittedfrom the server or any frame of the requested series of views stored onthe local data storage medium.

The method may include storing on the client device local data storagemedium one or more frames transmitted from the server of the requestedseries of views that were not previously stored on the local datastorage medium.

The volume visualization dataset may comprise a plurality ofcross-sectional images taken at spaced intervals through an object. Theclient device may request a series of three-dimensional virtual views ofat least a portion of the volume visualization dataset, with the seriesof views comprising a plurality of separate view frames. The requestedseries of three-dimensional virtual views of the volume visualizationdataset may be displayed at the client device by sequentially displayingframes transmitted from the server along with any frames of therequested series of views stored on the local data storage medium.During the displaying of the requested series of three-dimensionalvirtual views of the volume visualization dataset at the client device,frames stored on the local data storage medium may be displayed alongwith frames transmitted from the server.

The method may include providing a plurality of central data storagemedia, with each data storage medium containing all or a portion of thevolume visualization dataset, and may include selecting the data storagemedium from which the requested frames of the requested series of viewsfrom the volume visualization dataset are created and transmitted.

The determination of storage of any frame of the requested at least oneview of the volume visualization dataset on the local data storagemedium may be made by creating a unique identifiable key of a request bythe client device of a three-dimensional virtual view of the volumevisualization dataset, storing on the local data storage medium theunique identifiable key of a prior request by the client device of athree-dimensional virtual view, comparing the unique identifiable key ofa current request by the client device of a three-dimensional virtualview with a stored unique identifiable key of a prior request by theclient device of a three-dimensional virtual view, and determining ifvalues of the current and prior unique identifiable keys are equivalent.If the values are equivalent, the method may include displaying from thelocal data storage medium a stored frame of the prior request of thethree-dimensional virtual view. If the values are not equivalent, themethod may include displaying a frame transmitted from the server of thecurrent request by the client device of a three-dimensional virtualview. The method may further include associating the unique identifiablekey of a prior request by the client device of a three-dimensionalvirtual view with a stored frame of the prior request of thethree-dimensional virtual view. If the values are equivalent, the methodmay include determining location of the stored frame of the priorrequest of the three-dimensional virtual view on the local data storagemedium, and using such location to display the stored frame.

In a further aspect, the present invention is directed to a method forviewing at a client device a series of three-dimensional virtual viewsof a volume visualization dataset contained on at least one centralizeddatabase. The method comprises providing at least one central datastorage medium containing the volume visualization dataset, providing atleast one server in communication with the at least one centralizeddatabase and capable of processing the volume visualization dataset tocreate virtual views based on client request, and providing a clientdevice linked to the server and central storage medium over theInternet. The method then includes sending from the client device to theserver a request for a plurality of three-dimensional virtual views ofat least a portion of the volume visualization dataset. The plurality ofviews comprises a plurality of separate view frames, and the requestincludes a request for a lower image quality parameter for the framesand a request for a higher image quality parameter for the frames. Themethod also includes, at the server, creating the requested frames fromthe volume visualization dataset at the lower image quality parameter,and transmitting the lower image quality parameter frames to the clientdevice, and displaying at least a portion of the requested lower imagequality parameter frames at the client device. The method furtherincludes, at the server, creating the requested higher image qualityparameter frames from the volume visualization dataset, and transmittingthe higher image quality parameter frames to the client device, anddisplaying the requested higher image quality parameter frames at theclient device.

After transmitting the lower image quality parameter frames to theclient device, the server may transmit the higher image qualityparameter frames to the client device while the client device isdisplaying the lower image quality parameter frames. The client devicemay display the higher image quality parameter frames prior to or aftercompleting display of the lower image quality parameter frames. Theframes may be transmitted from the server to the client device as acompressed video stream, or as one or more single frames.

In yet another aspect, the present invention is directed to a method forviewing at a client device a series of three-dimensional virtual viewsof a volume visualization dataset contained on at least one centralizeddatabase. The method comprises providing at least one central datastorage medium containing the volume visualization dataset, providing aserver in communication with the at least one centralized database andcapable of processing the volume visualization dataset to create virtualviews based on client request, and providing a client device linked tothe server and central storage medium over the Internet. The method thenincludes requesting at the client device a series of three-dimensionalvirtual views of at least a portion of the volume visualization dataset,with the series of views comprising a plurality of separate view frames.The method also includes separating at the client device the requestedseries of three-dimensional virtual views into different groups offrames, and sending from the client device to the server a request for afirst group of frames, while delaying sending to the server a requestfor a second group of frames. The method further includes, at theserver, creating the requested first group of frames from the volumevisualization dataset, and transmitting the first group of frames to theclient device, and displaying the requested first group of frames at theclient device while simultaneously sending from the client device to theserver a request for a second group of frames. The method also includes,at the server, creating the requested second group of frames from thevolume visualization dataset, and transmitting the second group offrames to the client device while the client device is displaying thefirst group of frames, and displaying the requested second group offrames at the client device immediately following the first group offrames, to maintain proper sequential display of the requested series ofthree-dimensional virtual views.

There may be provided a plurality of servers in communication with theat least one centralized database and capable of processing the volumevisualization dataset to create virtual views based on client request,and a security device controlling the plurality of communicationsbetween a client device, and the server and central storage medium.

The method may further include sending from the client device to theserver a request for a subsequent group of frames while a previous groupof frames is being displayed, creating at the server the requestedsecond group of frames from the volume visualization dataset,transmitting the second group of frames to the client device while theclient device is displaying the previous group of frames; and displayingthe requested subsequent group of frames at the client deviceimmediately following the previous group of frames to maintain propersequential display.

The client device and server may be connected by a network having alatency time, and the requested groups of frames may be delayed insending from the client device to the server in accordance with thelatency time. The groups of frames are transmitted from the server tothe client device as a compressed video stream, or as one or moreindividual frames. The requested frames may be stored at the clientdevice local data storage medium.

The request for a first group of frames may also include a request for alower image quality parameter for the frames. After transmitting thefirst group of frames to the client device at the requested lower imagequality parameter, the server may transmit frames to the client deviceat a higher image quality parameter.

BRIEF DESCRIPTION OF THE DRAWINGS

The features of the invention believed to be novel and the elementscharacteristic of the invention are set forth with particularity in theappended claims. The figures are for illustration purposes only and arenot drawn to scale. The invention itself, however, both as toorganization and method of operation, may best be understood byreference to the detailed description which follows taken in conjunctionwith the accompanying drawings in which:

FIG. 1 is a schematic showing the major components and their interactionin operation of the present invention system and method for viewing at aclient computer a series of three-dimensional virtual views over theInternet of a volume visualization dataset contained on at least onecentralized database.

FIG. 2 shows the requests that may be made by the client computer fortransmission of frames of views of the volume visualization dataset atdifferent combinations of performance (transmission rate) and imagequality.

FIG. 3 is a flow diagram showing one embodiment of the method ofcombining local- and server-stored images of frames of the volumevisualization dataset.

FIG. 4 is a flow diagram showing one embodiment of the method ofreducing delays in displaying view frames transmitted from a centralserver and data storage due to network latency.

FIG. 5 is a flow diagram showing one embodiment of the method of rapidlydetermining if requested view frames are available on the user's localstorage medium.

DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

In describing the preferred embodiment of the present invention,reference will be made herein to FIGS. 1-5 of the drawings in which likenumerals refer to like features of the invention.

The present invention in one embodiment provides a method and system forfast access to advanced visualization of medical scans using a dedicatedweb portal, in particular, advanced visualization of volumetric data,for example, medical scans. As shown in FIG. 1, one embodiment of themethod and system 20 of the present invention consists of a number offunctional components or layers, i.e., Application Layer 30, ExternalNetwork Layer 40, Security Layer 80, Load Balancer Layer 90, and Storageand Processing Layer 100. Communication and transmission connectionsbetween the individual system components or layer, as well as with theserver and users, may be made by wireless, wireline, optical fibercable, radio frequency (RF) or the like, and may also be made via aninternal network (LAN), wide area network (WAN) and/or over the Internetor other connection means. The individual system components may beincorporated as hardware or software modules within a computer.Communication and transmission connections that permit a higher datatransmission rate are preferred, although the present invention isparticularly useful where low bandwidth and high latency situationsexist between the user and the volumetric data.

The customer or user uses an otherwise conventional web browserapplication 32 on a computer or other electronic device 22 at a remotelocation capable of communicating across an intranet or the Internet,which device also may have connection 23 with a local computer-readablestorage medium 24 that may be used as the customer's data source.Storage medium 24 may be employed to receive all or portions of volumevisualization dataset(s) such as CT or MRI scans made elsewhere, andfunction as a transmitter of such dataset(s) to a centralized database,as will be discussed further below. Storage medium 24 may also beemployed to receive and store downloads from the centralized database ofall or portions of volume visualization dataset(s) or views createdthere from. One or more computer readable medium(s) may be utilized,alone or in combination. A suitable computer readable storage medium maybe, for example, but not limited to, an electronic, magnetic, optical,electromagnetic, infrared, or semiconductor system, apparatus, ordevice, or any suitable combination of the foregoing. Other examples ofsuitable computer readable storage medium would include, withoutlimitation, the following: an electrical connection having one or morewires, a portable computer diskette, a hard disk, a random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or flash memory), an optical fiber, a portable compactdisc read-only memory (e.g., CD-ROM), an optical storage device (e.g.,DVD ROM), a magnetic storage device (e.g., diskette), or any suitablecombination of the foregoing. A suitable computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

At the Application Layer 30, there may be one or many web browserapplication(s) 30 and one or many onsite router(s) 34. The ApplicationLayer is connected to External Network Layer 40 that employs theInternet 50 or any other network that connects the local network in theapplication layer to the rest of the layers using an encrypted channelof communication (such as SSL/TLS). Secure communication ports may beused that communicate only with SSL/TLS turned on, for example, a443/8443/9443/custom port. This connection may be a high or lowbandwidth connection 51 between the web browser and the Internet, andmay include connection 54 between the customer data source 24 and onsiterouter 34, and connection 52 between the onsite router and the Internet.A virtual private network (VPN) 36 may also be used between the customerdata source and the Internet. The connection may be facilitated by oneor many domain name service (DNS) lookup services 42 and may be used toserve as a load balancer as well for directing traffic for when one ormany serving physical sites or servers within it are down.

The External Network Layer connects via connections 55 to one or manyserving sites 70, 70′ that comprise of Security Layer 80, Load BalancerLayer 90 and Storage and Processing Layer 100. The Security Layer mayconsist of one or many firewall(s) 76, intrusion detection system(s) 72,and intrusion prevention system(s) 74. Load Balancer Layer 90 maycontain at least two resource manager servers 92 with master and slaveconfiguration. Load Balancer Layer 90 manages the allocation andexpansion of servers to handle the scalability of storage capacity andprocessing power dynamically as Storage and Processing Layer 100communicates with the user 22 through the Security Layer. At the Storageand Processing Layer there may lay a cluster of servers 102 that havestorage capabilities and high performing vector processors. Servers 102communicate with Security Layer 80 via connections 57 and 58 throughresource manager 92, or directly via connection 59. The Security, LoadBalancer and Storage and Processing Layers may be located physically ata serving site 70. For redundancy, one or many duplicate Security, LoadBalancer and Storage and Processing Layers are located physically at oneor more additional serving sites 70′. Serving sites 70, 70′ may beconnected with each other via External Network Layer 40. Additionally oralternatively, Storage and Processing Layers 100 at disparate locations70, 70′ may be connected via a private network 60 via connections 56 sothat information exchange between them can be fast.

Storage and Processing Layer servers 102 may consist of server clustershaving one or more computer-readable volume data storage media (asdescribed above), volume processing server, and web or applicationserver for application interaction. The servers 102 are in communicationwith the computer-readable volume data storage mediums at the servingsites and are capable of processing volume datasets in the data storagemediums and communicating with the viewing applications of thevisualization of the processed information. The Storage and Processingservers may receive data transmitted from a user's storage medium 24 bymanual upload through the web browser application 30. Alternatively, theStorage and Processing servers may receive data from storage medium 24transmitted through an onsite router 34 installed at the data source oran offsite router 78 installed at the Security Layer 80, connected via asingle or bidirectional VPN 36, 136, respectively. The router 34, 78 maybe configured to access existing storage infrastructure at the sourcelocation, and may be configured to move the source data from the sourcelocation to the Storage and Processing servers. The data movement mayeither be done through manual methods by authorized personnel accessingthe router, or through automatic rules that may be setup prior to use.Rule criterion may be but not limited to moving data using a particulardate interval, or based on the volume type, or a volume data belongingto a particular user or institution, etc. A router negotiates with theload balancer layer to identify which storage device server is availableto accept a particular data. The criterion for selecting which storagedevice is available is determined but not limited to by the size of thevolume data and the capability of the server to accept the volume data.Once data is available on the volume data storage device servers,information about the data is entered to a database and may be presentedto user for visualization by the web portal.

Advanced visualization of volume datasets begin with identification ofthe volume datasets or subset of the datasets to visualize; specified asa user interaction parameter. The identification may include (a) thefull volume datasets, (b) one or many planar views through volumedatasets with each planar view having variable thickness, (c)curvilinear sampling of the volume datasets, (d) a region of interestsampling of the volume datasets, or (d) segmented volume datasets tosimplify and/or change the representation of the volume datasets intosomething that is more meaningful and easier to analyze. The identifiedset of volume dataset may be then visualized by (a) applying a color,opacity, gradient, and/or multidimensional based transfer function tothe intensity value of each individual voxels, and (b) using ray castingor texture based rendering techniques. Based on user interactionparameters, object transformation, projection transformation, clippingand screen transformation are applied to produce a rendered frame, alist of user interaction sequence generates one or many frames.

User 22 interactions to access advanced visualization of volumetric datavia a web portal may begin with a web application 30 for accepting auser or a program login and listing the volume data descriptionassociated with the login, then launching a viewing application that isexecutable through a web browser. The viewing application takes a useror program input and communicates with one or many Storage andProcessing servers 102, and displays the processed information as anadvanced visualization of the volume data. For example, a customer oroperator such as a physician, a patient, or a program logs in to volumevisualization web portal using a web browser and identifies as anauthorized user 22. A list of volume data such as a list of patient CTscans that is associated with the authorized user is presented to webapplication 30. The user may select one or more static or dynamicvirtual images from the volume visualization dataset and request to viewthe images on the web application.

Static 3D virtual images may be made by selecting a plane to be cutthrough the volume of the object at a particular angle, at the desiredlocation within the scanned volume of the object. Dynamic 3D virtualimages may be made by moving the selected view plane through the volumeof the scanned object, for example, by rotating it about a line, or bymoving its center along a path, or a combination of both. This resultsin a series of 3D virtual views. The series of views comprise pluralityof separate view frames created sequentially at different times duringthe movement of the view plane. Each frame contains the imageinformation from the portions of the original scans that it intersects,and may further contain image information interpolated between theoriginal scans. When the view frames are played back and displayed on ascreen in the same sequence, the user is able to view the requestedseries of 3D virtual views in the manner of viewing a video. A requestfor a 3D virtual image is referred to herein as an interaction, and themovement of the view plane is referred to herein as an interactionsequence.

One or more of the created frames may be compressed as a video streamand delivered to the viewing application. A video stream with more thanone frame can increase compression ratios because the compressionalgorithms can take advantage of the redundancy that exists not only inan intra-frame but in the inter-frame and temporal motion-compensationinformation of the many frames.

Following the user's interaction to select one or more views, a requestis initiated to Load Balancer 90 to identify the Storage and Processingservers 102 with which to interact. The Load Balancer server determineswhich Storage and Processing server 102 from the available cluster iscapable of servicing the user's request. The criterion for selectingwhich server is determined but not limited to by the size of the volumedata, the capability of the server to render the volume data, currentload of the server, and the location of the data. Once a server 102 isselected, the Load Balancer responds to the web application and the webapplication creates a viewing application with the appropriate creationparameter that links the viewing application to the Storage andProcessing servers. The viewing application may be a web application inthe form of an Active-X component, or a Java applet or a web browserscriptable language such as JavaScript, Flash or Silverlight. The webapplication may display the viewing application either as one or morenew web page with dynamic window size, or on full screen mode, or anembedded viewer on an existing or new web page, or launched as astand-alone viewer. The full screen mode displays visualization on theentire screen or desktop or on multiple monitors.

Once the application is initiated, user 22 makes a request to theStorage and Processing server 102 to obtain desired volume datainformation. The volume data information may contain but not limited toaspect ratio of voxel, voxel spacing, dimension and bit depth. Thisallows the application to prepare the appropriate initial presentationof the volume data to a user. The Storage and Processing server loadsthe data from its storage medium. The loading progress of the volumedata may be displayed to the user by having the volume visualization webserver send video streams and by having the application display thestream. The video stream may be a collection of frames generated by theStorage and Processing server and may be compressed either through knownlossless or lossy compression methods, depending on user preferences for(i) quality of visualization of processed images, and (ii) interactionframe rates. Depending on these two parameters and the network bandwidthand latency (discussed further below), the Storage and Processing servermay choose the appropriate compression type. The compression algorithmcan either be executed on central processing unit (CPU) or graphicsprocessing unit (GPU) at server 102. Use of GPU and machine-learningtechniques can be applied to enhance compression ratio and performance.

The compression parameters may be adapted but not limited to the imagequality requirements of the user, and network bandwidth size. Imagequality parameters may be set as an option on the application by theuser to adapt to different image quality or performance duringinteraction, and for when the interaction has stopped. A lower imagequality parameter for the frames may be transmitted over the Internetfrom the server 102 to the user device 22 at a faster rate than a higherimage quality parameter for the frames. As shown in FIG. 2, a request110 may be made by the user device 22 to limit frame resolution, forexample to 512×512 pixels and to stream the transmission of compressedframes while user interaction occurs. The balance of frame resolutionmay be set by the slide which ranges from high performance (transmissionrate) and low image quality at the left, to low performance(transmission rate) and high image quality at the right, or anycombination in between. FIG. 2 also shows that user device 22 may alsotransmit a request 120 to update the image quality, for example, whenthe user interaction stops, to a different combination of performance(transmission rate) and image quality. These options allow one to use alower quality image to facilitate delivery of high frame rates and onceinteraction is stopped, an image quality with highest fidelity may berestored. Once the application 30 receives the stream, it may decompressthe stream to retrieve individual frames contained within the videostream, and displays each frame successively at the user device with theappropriate timing.

The server may transmit the higher image quality parameter frames to theclient device while the client device is displaying the lower imagequality parameter frames, and the client device may display the higherimage quality parameter frames prior to or after completing display ofthe lower image quality parameter frames.

When the volume data is fully loaded or while the volume is beingloaded, user 22 is allowed to interact with the partially andprogressively loaded volume. Such interaction begins by the usermanipulating the application 30 user interface to perform functions suchas but not limited to orientation such as zoom, rotate, pan, crossreference, threshold adjustment, window/level adjustment, transferfunction manipulation, drag and drop to control the layout of multiplevolumes, fusion to manipulate the combining of two or many volumes,segmentation to remove parts of the volume or simplify the visualizationof the volume, supervision for machine learning input, and the like.

A user interaction sequence may be described as simple keyboard or mouseaction, or as a complex graphical state such as ones presented by OpenGLstandard, or combined with other peripheral information pertaining togenerating an advanced visualization of the volume dataset. Aninteraction sequence is encapsulated as but not limited to an XMLdocument. User interaction sequences may be communicated to Storage andProcessing server 102 using a standard inter-process communication (IPC)technology by invoking its appropriate procedure. In an IPC request, aserver 102 procedure name and parameters is encapsulated using but notlimited to an XML document. The procedure may be executed but notlimited to using either through a secure remote procedure call (RPC), orthrough an encrypted XML on hypertext transfer protocol (HTTP), orthrough a hypertext transfer protocol secure (HTTPS). The informationreturned to the user of a procedure may also be in the form of an XMLdocument. The IPC communication is secured by encrypting the channel;for example HTTPS protocol provide default SSL/TSL encryption whileother protocols may be encrypted using a public key and private keycombination. The user may exit the application using the application'sclose function, for example, by selecting a logout command on the webpage on which the user or program logged in initially.

To overcome the network latency between the application 30 and server102, the user interactions are queued up to the network latency time(usually less than 300-350 milliseconds) and then the collection of userinteractions are sent to the Storage and Processing server. As shown inFIG. 4, the method of reducing delays in displaying view framestransmitted from a central server 300 includes step 310, where a user'srequests for interaction are broken up into multiple requests that arequeued up at Application 30 for a time equal to network latency time(e.g., 350 ms). Next, in step 320, the requests are sent sequentially toserver 102. The Storage and Processing server generates images as framesfor each user interaction and generates a video stream from thecollection of frames. In step 330, the video stream made up of firstrequested frames may be compressed by server 102 either through losslessor lossy compression scheme and sent to application 30, for example, asa compressed video stream. The application may then play this videostream by decompressing the stream and displaying each frame at thecorrect timing (step 340). During the time the volume visualization webserver is rendering and sending the video stream back to theapplication, the user interaction sequences may be continuously acceptedby the application, and the interactions may be queued up according tothe network latency time as described previously and sent to the server(step 350). The process of gathering interaction sequence, sending thosesequences to the web server and playing the video stream of the previousinteraction sequences may be repeated until the user 22 stopsinteracting. The server continues to send compressed video streams fromsubsequent requests, which the application then displays to the user(step 360). By overlaying the collection of user interactions sequencesat the application end, and with the processing time for generating thevideo streams by server 102, the network latency and rendering time canbe virtually hidden. For example, assume that the network latency timeis 300 ms, and during that time, user interactions sequences are beingcollected. When the sequences are submitted at 300 ms time, the videostream of the previous sequence has now become available. Hence, afteran initial lag of 300 ms, the interaction may be made smooth because theapplication is continuously collecting new user interactions sequenceand playing back video streams of the previous sequence. The playback ofthe video stream needs may be done at the correct timing to make thevideo stream of images play seamlessly.

A user interaction and/or interaction sequences along with thecorresponding frame or video stream that has been obtained from theStorage and Processing servers may be tracked and kept in availablecomputer-readable storage medium 24 on the user's computer by theapplication. The frame may be kept as it is or in a compressed form;lossless compression may be preferred to retain the original imagequality but near lossless or lossy compression algorithms may also beacceptable. This enables the application to reuse the frame or videostream when a user performs the same interaction or interactionsequences without having to request it from the volume visualization webserver, thereby eliminating network traffic and increasing displayperformance.

The present invention provides a method of combining local- andserver-stored images of frames of the volume visualization dataset, asdepicted in FIG. 3. The method 200 provides that a user 22 may requestfrom application 30 at the client device an interaction, i.e., one ormore frames of three-dimensional virtual views of volume visualizationdataset (step 210). The application checks to see if any requestedframes are stored on local storage medium 24 (step 220). If so, theapplication displays the frames from the local storage medium. If allrequested frames are not available locally, the application sends arequest for unavailable frames to server 102 (step 240). The server thencreates the requested frame, i.e., a 3D view, from raw data, i.e., the2D scans stored in the volume visualization dataset in the centralstorage medium (step 250). The server then transmits the createdframe(s) to the application on the client device, e.g., as a videostream (step 260). The application then displays frames received fromthe server, optionally mixed with frames available from local storage(step 270).

Identification of such repeated user interactions and the correspondingstored frames or video stream should be done as rapidly as the sequenceof frames is to be displayed to the user. For example, the entire lookupand rendering have to be done at least within one-eighth of a second todeliver an acceptable 8 frame per second (FPS) interactivity anddisplay. One method 400 for rapid retrieval is to use a sortedassociative container in local storage medium 24, as depicted in theflow diagram of FIG. 5. When a 3D view frame is transmitted from theserver 102 to the user 22, a copy is stored in the user's local storagemedium 24. The local storage medium may include a container that hasstored frames in a record created by a previous request by the user of a3D view (step 410). In a sorted associative container, a record isobtained by providing to the container an identifiable key (step 420).Two records within such a container have two separate keys which areconsidered to be equivalent if neither one is less than the other (step430). Elements of a user interaction XML document can be compared withanother to determine whether the values of the elements in either oneare less than the other. So, a unique identifiable key may be made foruser interaction XML document. Presenting a user interaction sequence tothe container will result in the container returning either a match or afailure to match (step 440).

A match may contain the information of the location of the frameassociated with that particular user interaction, for example a memoryaddress of the frame or a file name of the frame (step 450). Where a webapplication is written in scripting language such as JavaScript, therecollection of rendered frames may be done by utilizing the webbrowser's caching mechanism. A browser may identify such image throughits uniform resource locator (URL) address. By providing the browserwith the same URL for a user interaction that has been executed before,the network communication to a server is eliminated. Thus when aninteraction sequence is created in response to a user's request, theinteraction sequence may be encapsulated as an XML document, and thisXML document may be used to query the associative container. If a matchis found, a round trip to server is avoided entirely and the framestored is used instead.

By keeping track of user interaction sequences and its correspondingrendered frames, bandwidth usage and server resources may besufficiently enhanced so that the interactive performance ofapplications in low bandwidth conditions is possible. For example, aRadiologist user may move from one Sagittal cross section to another(commonly known as “Stacking”). When the user returns to the previouscross section, the rendered frame is immediately retrieved withoutmaking a round trip to the server. Similarly, if a user does a 360degrees rotation at 10 degrees interval (commonly known as “360-view”),then after 36 requests to the server, all the frames needed to performsubsequent 360 rotation have already been obtained. This enables theuser to continue to rotate in either direction of the 360 degrees byusing only the frames stored on local storage medium 24, withoutapplication 30 needing to request the frames from server 102. Also, whenthe user interface is set to a specific tool, for example, Stacking or360-view, the application can predict all combinations of userinteraction sequence. A user may elect to automate the pre-fetching ofinteraction sequences when prediction by application is possible. Aftera short wait, all the frames are available for interaction for the givenset of interaction sequences. In summary, by keeping track of userinteractions parameters and the corresponding locally stored frames, andby having a rapid way to identify the current user interaction withprevious interaction parameters, as well as automating pre-fetching ofinteraction sequences, the present invention is able to achieveinteractivity over low bandwidth and high latency networks.

Keeping track of user interaction sequences and its correspondingrendered frames also enables the display of 4D or higher dimensionaldataset at interactive frame rates over low bandwidth. An element of auser interaction sequence is reserved to indicate which scan iscurrently being rendered. Suppose a cardiac scan of a patient contains atime series of 16 scans taken at various interval of the heart beatcycle. So, after iterating through the 16 scans and rendering each viewand with other elements of the user interaction sequence fixed, theplayback of the 16 scans results in a real-time animated sequence of theheart. This is because none of the 16 views require a round trip to theserver. This makes it possible to display interactively 4D or higherdimensional data over low bandwidth.

The Storage and Processing servers may render a volume visualizationdataset as described in U.S. patent application Ser. No. 11/672,581,publication no. US 2008/195843, for Method and System for Processing aVolume Visualization Dataset, filed on Feb. 8, 2007, the disclosure ofwhich is hereby incorporated by reference. Machine learning algorithmsmay be applied to the processing of volume visualization dataset, forexample, during segmentation of a volume to provide semi-automaticsegmentation through unsupervised and supervised learning techniques.Genetic algorithms, self organizing maps and neural networks may beutilized to process the volume visualization.

The Storage and Processing Layer 100 may have: 1) server hardwareequipped with one or multiple operating systems using zero or multiplevirtual machines, 2) one or many computer program(s) responsible foraccepting and responding to viewing application in accordance to HTTPand/or HTTPS, 3) one or many computer program(s) responsible foraccepting and responding to viewing application in accordance to UserDatagram Protocol, 4) one or many computer program(s) responsible foraccepting and responding to a network monitoring system, have one ormany central processing unit (CPU), 5) one or many co-processors, whichmay include field programmable gate arrays (FPGAs), vector processorslike the graphics processing unit (GPU), cell processors, orco-processors embedded in the same physical chip as the CPU, and/or 6)storage devices such as random access memory, flash memory, solid statedrive or hard disk, or any one or more of the additional above-mentionedcomputer readable storage mediums.

The entire cluster of servers 102 behind the firewall 76 may benetworked with high speed link such as GigE, 10 GigE or Infinibandswitch through but not limited to standard Category 5 cable or fiberoptic cables. This enables the servers to transmit or exchange databetween these servers at a high speed. The server clustered may beconnected 55 to the Internet using an external service provider such asAT&T network carrier by connecting the external link to the switch. Theserver clusters may be networked privately within carrier neutral datacenters so each servers can communicate on different sites rapidly. Anetwork monitoring system is utilized to identify the network forproblems caused by overloaded and/or crashed servers, networkconnections or other devices on the network.

For compliance with applicable regulations and for protecting theStorage and Processing servers clusters, the Security, Load Balancer andStorage and Processing Layers in each of serving sites 70, 70′ may beinstalled at a SAS70 Type II Compliant data center pursuant to periodicstatements issued by the Auditing Standards Board of the AmericanInstitute of Certified Public Accountants (AICPA). In such a datacenter, logs of user entering or leaving are recorded, the facility isphysically protected through but not limited to fingerprint, handprintor other biometric base identity entry system, the racks or cages arelocked and monitored via cameras, and a hardware firewall and intrusiondetection are installed and monitored to fend off cyber attacks. Onlyports that enable web methods are opened on the firewall. Examples ofstandard ports are 80 for HTTP or 443 for HTTPS. A private UDP port mayalso be opened to provide a better video streaming experience to theviewing application. An intrusion detection and prevention system 74 maybe used to detect several types of malicious behaviors that cancompromise the security and trust of a clustered system. This includesnetwork attacks against vulnerable services, data driven attacks onapplications, host based attacks such as privilege escalation,unauthorized logins and access to sensitive files, and malware (viruses,trojan horses, and worms). The communication between the web applicationand the Storage and Processing servers may be encrypted. A VPN accessthrough a function of the firewall device may be enabled for remotemonitoring and updating of cluster of servers behind the firewall. Fromwithin a VPN session, authorized remote desktop applications may beenabled to gain remote access to the servers.

The present invention may be embodied as a system, method or computerprogram product. The present invention may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” The presentinvention may take the form of a computer program product embodied inone or more computer readable medium(s) having computer readable programcode embodied thereon. Program code embodied on a computer readablemedium may be transmitted using any appropriate medium, including butnot limited to wireless, wireline, optical fiber cable, radio frequency(RF) or the like, or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

The present invention is described herein with reference to diagrams offunction blocks or modules in the drawings showing methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block and combinations ofblocks in the drawings may be implemented by computer programinstructions. These computer program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the function blocks ormodules in the drawings.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the function blocks or modulesin the drawings.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the function blocks ormodules in the drawings.

The function blocks or modules in the drawings illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in thedrawing may represent a module, segment, or portion of code, whichcomprises one or more executable instructions for implementing thespecified logical function(s). It should also be noted that, in somealternative implementations, the functions noted in the block may occurout of the order noted in the figures. For example, the function of twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block and combinations of blocks in the drawing may beimplemented by special purpose hardware-based systems that perform thespecified functions or acts, or combinations of special purpose hardwareand computer instructions. Also, although communication between functionblocks or modules may be indicated in one direction on the drawing, suchcommunication may also be in both directions.

The present invention therefore provides a processing and accessmechanism that overcomes low bandwidth and high latency limitations thatare inherent properties of the standard Internet and permits bandwidthusage to be optimized, particularly for interactive web application forvisualizing large medical scans on any Internet connection.

The present invention also overcomes the problems of maintaining medicalrecords for long periods of time under security and privacy by providinga method and system of a common and centralized infrastructure, forreceiving, storing, processing and viewing large medical scans via a webportal where economic of scale can be applied generously.

The present invention further provides fast access to imaging studiesand immediate 3D manipulation of the dataset over a standard, lowbandwidth, Internet connection. The present invention permits one tocarry out one or more of the following specific aims to empowerradiologists, clinicians, surgeons and patients to access medical scans:A HIPAA compliant and secure web portal for fast access by physiciansand patients enables sharing of medical scans between participatinghealth care institutions' systems using DICOM and HL7 standards. Also,patients are able to upload their scan from a CD or by providing theirconsent to obtain the scans from non-participating institutions on thepatient's behalf. This enables hospitals to interface with anyphysicians or specialist from anywhere, allows the physicians tocommunicate among themselves and with patients effectively, and enablespatients, especially in rural areas, to have access to any physiciansanywhere to provide proper diagnosis or second opinions. The portal isbased on centralized and high performing vector processor basedclustered servers for generating easy to comprehend 2D, 3D or 4Dintelligent visualization of large scans and transmitting those views toend users.

While the present invention has been particularly described, inconjunction with a specific preferred embodiment, it is evident thatmany alternatives, modifications and variations will be apparent tothose skilled in the art in light of the foregoing description. It istherefore contemplated that the appended claims will embrace any suchalternatives, modifications and variations as falling within the truescope and spirit of the present invention.

Thus, having described the invention, what is claimed is:
 1. A systemfor viewing at a client device at a remote location a series of virtualviews over the Internet of a medical imaging dataset contained on aremote database, comprising: at least one remote data storage mediumcontaining the medical imaging dataset; at least one server incommunication with the remote database and capable of processing themedical imaging dataset to create virtual views based on a clientrequest; at least one physically secured site for housing the at leastone server, the secured site comprising an audited or tested premises;and a web application adapted to satisfy a user's request for thevirtual views by: a) accepting at a remote location via an encryptedcommunication connection at least one user request for a series ofvirtual views of the medical imaging dataset, the series of viewscomprising a plurality of separate view frames, the remote locationhaving a local data storage medium for storing frames of views of themedical imaging dataset, b) transmitting from the remote location to theat least one server via the encrypted communication connection a requestfor any frame of the requested views not stored on the local datastorage medium, c) at the at least one server, creating the requestedframes of the requested views from the medical imaging dataset in theremote storage medium, d) transmitting, via the encrypted communicationconnection, the created frames of the requested views from the at leastone server to the client device, e) receiving, and storing, on the localdata storage medium the requested views from the at least one server,and f) displaying to the user at the remote location the requestedseries of virtual views of the medical imaging dataset by sequentiallydisplaying frames transmitted from the at least one server along withany frames of the requested series of views stored on the local datastorage medium.
 2. The system of claim 1 wherein the web application isadapted to select a secured site by using a domain name address on a webbrowser.
 3. The system of claim 1 wherein the web application is adaptedto be authenticated using an authorized user credentials prior totransmitting the request to the at least one server and lists themedical imaging datasets that are viewable by the authorized user. 4.The system of claim 1 wherein the web application initiates a virtualviewing by selecting a medical imaging dataset.
 5. The system of claim 1wherein a plurality of physically secured sites are located in disparatephysical locations and are connected via an Internet or an Intranet. 6.The system of claim 5 wherein the plurality of physically secured sitesoffer distributed data processing capabilities.
 7. The system of claim 1wherein the remote database or remote data storage medium offercentralized or distributed storage, processing, or retrievalcapabilities.
 8. The system of claim 1 wherein the web applicationcommunication with the at least one server is via a web applicationprogramming interface.
 9. The system of claim 1 wherein, for each viewrequest, the web application determines location of the stored frame ofthe prior request of the virtual view on the local data storage medium,and uses such location to display the stored frame if one exists. 10.The system of claim 1 wherein the web application causes the remotelocation to display higher image quality parameter frames aftercompleting display of lower image quality parameter frames.
 11. Thesystem of claim 1 wherein the web application causes the server totransmit the frames to the remote location as a compressed video streamor as one or more single frames.
 12. The system of claim 1 whereingenetic algorithms, self-organizing maps, neural networks or othermachine learning algorithms are applied during processing of the medicalimaging dataset.
 13. A method for viewing at a client device at a remotelocation a series of virtual views over the Internet of a medicalimaging dataset contained on a remote database, comprising: providing atleast one remote data storage medium containing the medical imagingdataset; providing at least one server in communication with the remotedatabase and capable of processing the medical imaging dataset to createvirtual views based on client request, a first or a secondary securitydevice controlling the plurality of communications between a clientdevice and the server; providing at least one physically secured sitefor housing the at least one server, the secured site comprising anaudited or tested premises; providing a web application accessible atthe client device linked to the at least one server and storage mediumover the Internet via an encrypted communication connection, the clientdevice having local data storage medium for storing frames of views ofthe medical imaging dataset, the web application adapted to perform thesteps of: requesting at the client device via the encryptedcommunication connection a series of virtual views of at least a portionof the medical imaging dataset, the series of views comprising aplurality of separate view frames; determining if any frame of therequested views of the medical imaging dataset is stored on the localdata storage medium; sending from the client device to the server viathe encrypted communication connection a request for any frame of therequested views; at the server, creating the requested frames of therequested views from the medical imaging dataset; transmitting thecreated frames of the requested views from the server to the clientdevice via the encrypted communication connection; and displaying therequested series of virtual views of the medical imaging dataset at theclient device by sequentially displaying frames transmitted from theserver along with any frames of the requested series of views stored onthe local data storage medium.
 14. The method of claim 13 wherein thereceived frames of the requested views are stored on the client devicelocal data storage medium.
 15. The method of claim 13 wherein, for eachview request, the client device determines location of the stored frameof the prior request of the virtual view on the local data storagemedium, and uses such location to display the stored frame.
 16. Themethod of claim 13 wherein the client device displays higher imagequality parameter frames after completing display of lower image qualityparameter frames.
 17. The method of claim 13 wherein the frames aretransmitted from the server to the client device as a compressed videostream or as one or more single frames.
 18. The method of claim 13further including the step of: applying genetic algorithms,self-organizing maps, neural networks or other machine learningalgorithms during processing of the medical imaging dataset.